Determining an interest level for an image

ABSTRACT

A method for determining an interest level of a digital image to a particular person. The digital image, or metadata associated with the digital image, is analyzed to designate one or more image elements in the digital image. Familiarity levels of the designated image elements to the particular person are determined. The interest level of the digital image to the particular person is then determined responsive to the determined familiarity levels. In some embodiments the image elements include persons and scene contexts, where digital images containing more familiar persons and less familiar scene contexts correspond to higher interest levels.

CROSS-REFERENCE TO RELATED APPLICATIONS

Reference is made to commonly-assigned, co-pending U.S. patentapplication Ser. No. 13/537,100, entitled: “Method for presentinghigh-interest-level images”, by Fedorovskaya et al.; tocommonly-assigned, co-pending U.S. patent application Ser. No.13/537,105, entitled: “System for presenting high-interest-levelimages”, by Fedorovskaya et al.; to commonly-assigned, co-pending U.S.patent application Ser. No. 13/537,099, entitled: “Modifying digitalimages to increase interest level”, by Fedorovskaya et al.; tocommonly-assigned, co-pending U.S. patent application Ser. No.13/537,106, entitled: “System for modifying images to increaseinterestingness”, by Fedorovskaya et al.; and to commonly-assigned,co-pending U.S. patent application Ser. No. 13/537,112, entitled:“Individualizing generic communications”, by Fedorovskaya et al., eachof which is incorporated herein by reference.

FIELD OF THE INVENTION

This invention pertains to the field of digital image analysis andorganization, and more specifically to a method for automaticallydetermining an interest level of a digital image to a particular person.

BACKGROUND OF THE INVENTION

Digital imaging has vastly increased people's ability to amass verylarge numbers of still images, video image sequences, and multimediarecords, combining one or more images and other content, for theirpersonal collections. (Still images, video sequences, and multimediarecords are referred to collectively herein by the terms “image records”or “images”, as appropriate.)

At the same time, with the pervasiveness of digital media, the use ofdigital images in computing, especially in human-computer interaction(HCI) for user interfaces and design, as well as in such wide-rangingareas as education, social media, art, science, advertising, marketingand politics, is rapidly becoming more and more significant. All ofthese applications present challenges to the organization, selection andretrieval of the most appropriate images for any given purpose.

Efforts have been made to aid users in organizing and utilizing imagerecords by assigning metadata to individual image records. Some types ofmetadata provide an indication of the expected value to the user. Forexample, many online databases and photo sharing communities allow usersto designate images as favorites by selecting appropriate tags andlabels, or to assign a rating for photos, such as an image qualityrating or an aesthetic appeal rating, or to otherwise express theiropinions by writing notes, issuing virtual awards and invitations tospecial user groups. An online photo-enthusiast community, Flickr, forexample, introduced the selection of the most interesting images for anypoint in time, wherein the “interestingness” is determined byconsidering several aspects associated with images including “click”statistics, the presence/absence of comments, favorite tags, and whomade them. In some applications, “favorite” tags or other comparabletags, (e.g. Facebook's “like” tag) are counted to provide a type ofpopularity ranking. The DPCchallenge and the Photobucket photo sharingsites encourage users to rate images on overall quality on a scale of 1to 10 through contests and challenges. By doing so, all these databasesallow users to efficiently access the best or most popular images. Manyof these photo sharing websites cater to photo-enthusiasts, amateurphotographers, or even professional photographers who attempt to captureand create unique and artistically looking images. They often chooseunusual subject matter, lighting, and colors or create specific effectsby editing their images with various creative and photo editing tools.Other online photo storage and sharing services, such as Kodak Gallery,Shutterfly, and Picasa, are primarily intended to serve consumers whocapture and share snapshots of everyday events and special moments withfamily and friends.

Social networking sites, such as Facebook, enable users to collectivelyaccumulate billions of images as a means of keeping in touch withfriends. Users can upload their photos and share them with friends, aswell as create prints, photo books and other photo-related items.Similar to online photo sharing communities, these services allow usersto selectively mark images as favorites, for example, by using the“like” tag, and to create other tags and annotations. In addition topictures, users increasingly upload and share video snippets, videofiles and short movies. YouTube is one of the most prominent examples ofa video sharing and publishing service, wherein users can upload videofiles in the form of videos, short movies and multimedia presentationsto share personal experiences, broadcast multimedia information foreducation purposes, and promote specific services/products. However,compared to the relative abundance of tags and rankings in photos sharedby communities of photo-enthusiasts and public and commercial image andmultimedia databases, tags and rankings are used considerably lessfrequently for images of friends and family. This limits theirapplicability for efficient image organization and retrieval.

To assist users in selecting and finding the best or most suitableimages, various methods have been developed. Typically, these methodsanalyze and evaluate subject matter categories, locations, scene types,faces of people in the photo and their identities, and other imageattributes extracted directly from image data or associated metadata forimage organization and retrieval purposes. For example, the article“Inferring generic activities and events from image content and bags ofgeo-tags” (Proc. 2008 International Conference on Content-based Imageand Video Retrieval, pp. 37-46, 2008) by Joshi et al. describes a methodfor classifying an image into a plurality of activity/event scenecategories in a probabilistic framework by leveraging image pixels andimage metadata.

The article by Yanagawa et al., entitled “Columbia University's baselinedetectors for 374 LSCOM semantic visual concepts” (Columbia UniversityADVENT Technical Report #222-2006-8, 2007) describes an activity/eventclassification method where image pixel information is analyzed usingsupport vector machine (SVM) based classifiers. These classifiers useimage color, texture, and shape information to determine anactivity/event classification for an image. In a related method, GPSmetadata associated with the images can be leveraged to obtain locationspecific geo-tags from a geographic database. Subsequently, a bag ofwords model can be combined with the SVM data to provide an improvedactivity/event classification.

While the organization and retrieval of images based on imageunderstanding and semantic analysis can be useful in some applications,selection of images based on subjective attributes, such as imagequality, user preference, subjective importance, and predictedaesthetic/emotional value is valuable to enable users to quickly accessthe best and/or most popular images in a collection. For example, U.S.Pat. No. 6,671,405, to Savakis et al., entitled “Method for automaticassessment of emphasis and appeal in consumer images,” discloses amethod for automatically computing a metric of “emphasis and appeal” ofan image without user intervention. A first metric is based upon anumber of factors, which can include: image semantic content (e.g.,detected people, faces); objective features (e.g., colorfulness,sharpness, overall image quality); and main subject features (e.g., mainsubject size). A second metric compares the factors relative to otherimages in a collection. The factors are integrated using a trainedreasoning engine. U.S. Patent Application Publication 2004/0075743, toChatani, entitled “System and method for digital image selection,” usesa similar method to perform image sorting based upon user selectedparameters of semantic content or objective features in the images.

Commonly-assigned U.S. Patent Application Publication 2003/0128389, toMatraszek et al., entitled “Method for creating and using affectiveinformation in a digital imaging system cross reference to relatedapplications,” discloses another approach that provides a measure ofimage record importance (i.e., “affective information”), which can takethe form of a multi-valued metadata tag. The affective information canbe manually entered by a user. It can also be automatically detected bymonitoring user reactions (e.g., facial expressions or physiologicalresponses), or user initiated utilization of a particular image (e.g.,how many times an image was printed or sent to others via e-mail). Theresulting affective information can be stored as metadata associatedwith a particular user. The use of affective metadata is generallylimited in that it requires exposure and accumulation of tags withrespect to already viewed images and does not directly translate tonovel, unseen, or untagged image content.

Commonly-assigned U.S. Pat. No. 7,271,809 to Fedorovskaya et al.,entitled “Method for using viewing time to determine affectiveinformation in an imaging system,” discloses a method for providingimage metadata based on image viewing time. With this approach, the timeintervals during which the user chooses to view each of the stilldigital images on the electronic displays are electronically monitoredand used to determine the degree of interest for each image.Subsequently, the metadata can be used to assist in retrieving one ormore images.

Commonly-assigned U.S. Pat. No. 8,135,684, to Fedorovskaya et al.,entitled “Value index from incomplete data,” describes another methodthat includes combining data about an image from multiple sources. Thedata that is combined includes capture related data, intrinsic imagedata (e.g., image quality data and image content data) and image usagedata, and is used to generate value indices for the images, which canthen be used to manage image sets.

Considering the very large numbers of image records, the rapid expansionof social networks and shared social media, and the increasing range ofapplications, there is a growing need for new and improved image andmultimedia selection methods. These new methods should take intoconsideration how users will respond to the selected content, even if itis novel and untagged. Preferably, the methods should determine whethera user will find an image interesting, and worthy of their attention. Inthis regard, research in psychology, neuroscience, communication andadvertising is providing useful information with respect to the natureof people's preferences, interests and reactions to objects andsituations, including complex imagery, and to the underlying perceptualand cognitive processing. This information can be utilized in developingalgorithms and methods for rating and selecting images and multimediacontent suitable for personal usage, as well as for visualcommunication, persuasion, advertising and other uses.

Photographs are not mere artifacts, but represent semiotic systems, fromwhich viewers derive meaning. As discussed by Scott in the article“Images in Advertising: The Need for a Theory of Visual Rhetoric”(Journal of Consumer Research, Vol. 21, pp. 252-273, 1994), people drawon accumulated past experiences in order to make sense of photographs.Although they may be initially attracted to an image because of itsquality, aesthetic properties, or low-level features, it has been foundthat viewers subsequently determine what is worthy of longer study basedon the potential that they see in the image of generating deepermeaning.

It has been found that there is a link between what people findinteresting and their familiarity with respect to the communicatedinformation. Unlike “recollection,” which entails consciously“remembering” an item, familiarity spurs a form of associativerecognition and has been explained as arising when “fluent processing ofan item is attributed to past experience with that item” (see:Yonelinas, “The Nature of Recollection and Familiarity: A Review of 30Years of Research.” Journal of Memory and Language, Vol. 46, pp.441-517, 2002). Familiarity has been defined and measured in two ways.Familiarity with an item's meaning involves the amount of perceivedknowledge a person has about an item or its meaningfulness to thatperson. Familiarity with regards to frequency of exposure is measured byhow often a person encounters the item.

The concept of “interestingness” (or equivalently “interest level”) hasbeen the subject of multiple interpretations. Interestingness has beeninterpreted as the attribute of an item, as the response of a user to anitem, as an emotion, or simply as a psychological or behavioralreaction. Vaiapury et al., in the article “Finding Interesting Images inAlbums using Attention” (Journal of Multimedia, Vol. 3, pp. 2-13, 2008),specify interestingness as “an entity that arises from interpretationand experience, surprise, beauty, aesthetics and desirability”, aprocess based on “how one interprets the world and one's accumulation ofexperience as embodied in the human cognition system”.

Interestingness has also been commonly equated to attention. Forexample, Katti et al., in the article “Pre-attentive Discrimination ofInterestingness in Images” (2008 IEEE International Conference onMultimedia and Expo, pp. 1433-1436, 2008), describe interestingness as“an aesthetic property that arouses curiosity and is a precursor toattention.”

Interest level has been put forward not only as a reaction of thecognitive system to stimulus, but has also been studied as an emotion(for example, see: Silvia, “What Is Interesting? Exploring the AppraisalStructure of Interest” (Emotion, Vol. 5, No. 1, pp. 89-102, 2005). Apartfrom the variables of novelty, complexity and surprise, “personalconnection” and “thought-provoking” have been identified as attributesthat contribute to the interestingness of pictures (for example, see:Halonen et al., “Naturalness and interestingness of test images forvisual quality evaluation,” Proc. SPIE 7867, 78670Z, 2011).

There remains a need for incorporating measures of familiarity intomethods for evaluating the interest level of images or multimedia itemsorder to improve ways of selecting information that can personallyappeal to the viewers and users of various multimedia collections,online communities, social networks and databases.

SUMMARY OF THE INVENTION

The present invention represents a method for determining an interestlevel of a digital image to a particular person, comprising:

automatically analyzing the digital image or metadata associated withthe digital image to designate one or more image elements in the digitalimage;

using a data processor to automatically determine familiarity levels ofthe designated image elements to the particular person;

determining the interest level of the digital image to the particularperson responsive to the determined familiarity levels; and

storing an indication of the determined interest level in aprocessor-accessible memory;

wherein the method is performed at least in part using a data processor.

This invention has the advantage that the interest level of a digitalimage to a particular person can be automatically determined without theneed for any user evaluation.

It has the further advantage that many different types of informationcan be used to determine familiarity levels and interest levels, and canadaptively take advantage of more information as it becomes available.

It has the additional advantage that digital images can be selected ormodified in order to provide customized high-interest-level images forapplications such as advertising, education and entertainment. Thehigh-interest-level images will be more likely to capture and maintainthe attention of the particular person.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level diagram showing the components of a system fordetermining an interest level of a digital image according to anembodiment of the present invention;

FIG. 2 is a flowchart of a method for determining an interest level of adigital image to a person of interest according to an embodiment of thepresent invention;

FIG. 3 shows more detail of the generate familiarity data for person ofinterest step in FIG. 2;

FIG. 4 shows more detail of the generate familiar contexts data step inFIG. 3;

FIG. 5 shows more detail of the generate familiar persons data step inFIG. 3;

FIG. 6 shows an example of a relational database for storing personalrelationship data;

FIG. 7 is a flowchart of a generalized method for determining aninterest level of a digital image to a person of interest according toan embodiment of the present invention;

FIG. 8 illustrates a user interface for collecting interest level datafrom a user;

FIG. 9A is a graph illustrating the relationship between scene contextfamiliarity and interestingness;

FIG. 9B is a graph illustrating the relationship between personfamiliarity and interestingness;

FIG. 10 is a flowchart of a method for displaying images having a highinterest level to a particular user according to an embodiment of thepresent invention;

FIG. 11 is a flowchart of a method for modifying an image to increasethe interest level to a particular user according to an embodiment ofthe present invention;

FIG. 12A illustrates an album page constructed using high-interest-levelimages according to an embodiment of the present invention; and

FIGS. 12B and 12C illustrates systems for presenting high-interest-levelimages to a particular user according to embodiments of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, some embodiments of the present inventionwill be described in terms that would ordinarily be implemented assoftware programs. Those skilled in the art will readily recognize thatthe equivalent of such software may also be constructed in hardware.Because image manipulation algorithms and systems are well known, thepresent description will be directed in particular to algorithms andsystems forming part of, or cooperating more directly with, the methodin accordance with the present invention. Other aspects of suchalgorithms and systems, together with hardware and software forproducing and otherwise processing the image signals involved therewith,not specifically shown or described herein, may be selected from suchsystems, algorithms, components, and elements known in the art. Giventhe system as described according to the invention in the following,software not specifically shown, suggested, or described herein that isuseful for implementation of the invention is considered conventionaland within the ordinary skill in such arts.

The invention is inclusive of combinations of the embodiments describedherein. References to “a particular embodiment” and the like refer tofeatures that are present in at least one embodiment of the invention.Separate references to “an embodiment” or “particular embodiments” orthe like do not necessarily refer to the same embodiment or embodiments;however, such embodiments are not mutually exclusive, unless soindicated or as are readily apparent to one of skill in the art. The useof singular or plural in referring to the “method” or “methods” and thelike is not limiting. It should be noted that, unless otherwiseexplicitly noted or required by context, the word “or” is used in thisdisclosure in a non-exclusive sense.

FIG. 1 is a high-level diagram showing the components of an imageanalysis system 50 that can be used for implementing an embodiment ofthe present invention. In a preferred embodiment, the image analysissystem 50 includes at least one user access device 30, at least oneimage server 10, and at least one social network server 20, connectedand accessible via a communications connection 40. For example, theimage server 10 and the social network server 20 can be accessible fromthe user access device 30 over the internet by means of a UniformResource Locator (URL). Alternately, the communications connection 40can be supplied by way of a personal network or intranet or anotherprivate or non-public digital network system such as a cellulartelephone network. Alternately, one skilled in the art will recognizethat the image analysis system 50 can operate over a distributedcollection of servers or file-sharing programs and devices.

Although the image server 10, the social network server 20 and thecommunications connection 40 are shown separately from the user accessdevice 30, one skilled in the art will appreciate that one or more ofthese may be stored completely or partially within the user accessdevice 30. The communications connection 40 is intended to include anytype of connection, whether wired or wireless, between devices, dataprocessors, or programs in which data may be communicated. Thecommunications connection 40 is intended to include a connection betweendevices or programs within a single data processor, a connection betweendevices or programs located in different data processors, and aconnection between devices not located in data processors at all.

In the embodiment of FIG. 1, the image server 10 includes a processor 11(e.g., a central processing unit (CPU)) for executing instructions and anetwork interface unit 12 for interfacing with the communicationsconnection 40. The image server 10 further includes one or moreprocessor-accessible non-volatile memory systems 13, such as hard diskdrives, for storing a plurality of digital images and facial dataprovided by the users, and one or more processor-accessible volatilememory systems 14, such as RAM, for use by the processor 11, the networkinterface unit 12, or by other system components.

The image server 10 also includes a database system 15 for storinginformation, including a user database for storing user information fora plurality of users. The user information can include user accessinformation such as a username and password. The database system 15further includes one or more databases for managing the plurality ofdigital images and facial data, together with metadata associated withthe digital images and facial data. The database system 15 may includeinformation written on the non-volatile memory system 13.

In the embodiment of FIG. 1, the social network server 20 includes aprocessor 21 for executing instructions and a network interface unit 22for interfacing with the communications connection 40. The socialnetwork server 20 further includes one or more processor-accessiblenon-volatile memory systems 23, such as hard disk drives, for storing aplurality of social relationship data pertaining to the users, and oneor more processor-accessible volatile memory systems 24, such as RAM,for use by the processor 21, the network interface unit 22, or by othersystem components.

The social network server 20 also includes a database system 25 forstoring information, including a user database for storing userinformation for a plurality of users. The user information can includeuser access information such as a username and password. The databasesystem 25 further includes one or more databases for managing theplurality of social relationship data pertaining to the users. Thedatabase system 25 may include information written on the non-volatilememory system 23.

One skilled in the art will understand that the user databases, as wellas other databases such as image databases, facial databases and socialrelationship databases could be constructed from a single database orfrom a plurality of databases. The databases can be stored on a singlehard drive or multiple hard drives, or can operate on the one or moreservers. Preferably the databases operate using Structured QueryLanguage (SQL), which is presently available in many commercialproducts, such as the MySQL software, or the like. Alternately, oneskilled in the art can construct database search and retrievalfunctionality in custom software and can store the database informationin one or more computer files. Such custom data storage and retrievalprograms are considered a type of database for purposes of the presentinvention.

A user is enabled to access the image analysis system 50 by way of useraccess device 30. A wide variety of user access devices 30 that arecurrently available can be employed for accessing the image analysissystem 50, including a personal computer (PC) or tablet computer with amodem or network card, a mobile phone with internet access, a digitalcamera device with internet access, a digital photo frame with internetaccess, a video gaming console with internet access, a set-top box ordigital media player device with internet access, or the like. The useraccess device 30 preferably includes a processor 31 for executinginstructions, a volatile memory system 34 for use by the processor, anetwork interface unit 32, an input device 35 (e.g., a mouse, akeyboard, a touch screen, a remote control, a pointer device or thelike, including a device that can accept inputs produced by gestures,body, head and eye movements, voice, bio-electric signals and othernon-contact user generated inputs) and a display device 36 (e.g., an LCDscreen, an LED screen, or display output connection for outputting to anexternal display device). The user access device 30 may also optionallyinclude an image sensing unit 37, such as a digital camera unit having aCMOS or CCD type image sensor array, for capturing digital images aswell as a non-volatile memory system 33 such as a flash memory ormagnetic hard disk or the like for storing digital image files, facialdata, and social relationship data.

In a preferred embodiment, each user is provided a user account on theimage analysis system 50 having an associated user name and password.The user is considered to be the owner of this account and is providedwith privileges to specify account settings for that account. The useris also provided with access to information, such as the digital images,associated with the accounts owned by other users as will be describedin further detail below. The inclusion of user accounts represents oneexemplary privacy control technique that can be used in combination withthe inventive method.

FIG. 2 is a flowchart showing a method for determining an interest level260 for a digital image 201 to a particular person of interest 301according to a preferred embodiment. Within the context of the presentinvention “interest level” is synonymous with the term“interestingness.” In the article “Beauty and interestingness” (Journalof Philosophy, Vol. 49, pp. 261-273, 1952) Haserot describesinterestingness as the “power of an object to awaken responses otherthan those called forth by its aesthetic form . . . . Interestingnessgenerates a certain kind of significance. Form gives beauty;interestingness gives emotional or conceptual meaningfulness”. A userresponse of interestingness can be distinguished from an “orientingresponse,” which is considered to be a reflex, involuntary reaction to anovel stimulus. Examples of an orienting response include turningsomeone's head or gaze to an abruptly appearing stimulus, such as asound, flicker or object motion. Interestingness can be defined as apower of an object, such as an image or a multimedia item, to elicit aresponse from the viewer that results in a sustained attention that goesbeyond orienting response, which is transient.

The digital image 201 can be any type of digital image file, such as adigital still image or a digital video, or a multimedia record. Thedigital image 201 is inclusive of one or more images in any combinationwith sounds or other data. Discussion herein is generally directed todigital images 201 that are captured using a digital still camera or adigital video camera, including devices such as camera phones orcomputers that incorporate a web camera. Digital images 201 can also becaptured using other capture devices such as digital scanners, or can becomputer generated images or graphics formed using a computerizedsystem. General features of digital still and video cameras, digitalscanners and computerized systems, together with algorithms forprocessing images provided by such devices, are well known, and thepresent description is generally limited to those aspects directlyrelated to the method of the invention. Other aspects, not specificallyshown or described herein may be selected from such systems, algorithms,components, and elements known in the art.

The digital image 201 includes one or more digital image channels orcolor components. Each digital image channel includes a two-dimensionalarray of image pixels, generally arranged by rows and columns, eachimage pixel having an associated pixel value. Each pixel value relatesto a signal level (e.g., a light level) for the corresponding digitalimage channel. For color imaging applications, the digital image 201will typically include red, green, and blue digital image channels.Digital videos include a time sequence of individual digital imageframes.

The calculation of the interest level 260 follows two paths. A firstanalysis path is used to calculate a context familiarity score 220providing an indication of the familiarity of the scene context for thedigital image 201 to the person of interest 301. A second analysis pathis used to calculate a person familiarity score 240 providing anindication of the familiarity of any persons pictured in the digitalimage 201 to the person of interest 301. A calculate interest level step250 then determines the interest level 260 responsive to the contextfamiliarity score 220 and the person familiarity score 240.

In the first analysis path on the left side of FIG. 2, a determine scenecontexts step 211 is used to determine image contexts data 212 providingan indication of one or more scene contexts for the digital image 201.The scene contexts can include information about the surroundings,environment or venue in which the image was captured. For example, thescene context can include information about a location for the scene,such as a generic location (e.g. office, beach, church) or a specificgeographic location (e.g. Yellowstone Park, the Eiffel Tower, theSerengeti plains). The scene context can also include information aboutan event or activity associated with the scene (e.g. birthday party,graduation ceremony, worship service, baseball game). The scene contextcan also include other types of information associated with the contextof the digital image 201, such as the season (e.g., winter) or time ofday (e.g., sunset). The scene contexts can also include sceneclassifications determined using any scene classification algorithmknown in the art. Examples of scene classifications can include indoor,outdoor, manmade, natural, urban and rural. The scene contexts can alsoinclude scene attributes of the scene, such as color statistics (e.g., adominant hue, average colorfulness, average brightness level, presenceor dominance of certain colors).

In some cases, a single image can be associated with multiple scenecontexts (e.g., a particular image can be associated with both a“birthday party” scene context and a “back yard” scene context). Thedetermine scene contexts step 211 can determine the scene context(s) forthe digital image 201 using any method known in the art. In a preferredembodiment, the determine scene contexts step 211 uses a feature-basedscene detection algorithm to automatically analyze the pixel data forthe digital image 201 to determine image features which can be used asinputs to a trained classifier that determines a scene context. Thefeature-based scene detection algorithm can be trained to directlyidentify scene contexts from a set of predefined scene contexts, such as“birthday party,” “back yard,” “office,” “train station,” etc. In someembodiments, the determine scene contexts step 211 uses thefeature-based scene detection algorithm described by Xiao et al., in thearticle entitled “SUN database: Large-scale scene recognition from abbeyto zoo” (Proc. IEEE Conference on Computer Vision and PatternRecognition, pp. 3485-3492, 2010), which is incorporated herein byreference.

In some embodiments, the determine scene contexts step 211 can determinethe scene context by using an object-based scene detection algorithm tofind and identify objects in the digital image. The presence of certaindetected objects can then be used to infer the scene context. Forexample, the presence of a birthday cake object can be used to inferthat an appropriate scene context for the digital image 201 would be“birthday party.”

In some embodiments, the determine scene contexts step 211 can determinethe scene context by comparing the digital image 201 to a set of labeledscene context reference images. In this case, the scene context can bedetermined in a non-parametric way by finding the closest matchesbetween the digital image 201 and the labeled scene context referenceimages. One method of doing this is described by Torralba et al. in thearticle entitled “80 Million tiny images: a large data set fornonparametric object and scene recognition” (IEEE Transactions onPattern Analysis and Machine Intelligence, Vol. 30, pp. 1958-1970,2008), which is incorporated herein by reference.

In some embodiments, the determine scene contexts step 211 can combinethe results of a plurality of analysis methods. For example, thedetermine scene contexts step 211 can apply both a feature-based scenedetection algorithm and an object-based scene detection algorithm, andcan combine the results using any method known in the art.

In a compare image context to familiar contexts step 213, the imagecontexts data 212 for the digital image 201 is compared to previouslycalculated familiar contexts data 380 to generate a context familiarityscore 220. The familiar contexts data 380 is a collection of informationabout scene contexts that are familiar to the person of interest 301.The familiar contexts data 380 is determined in a generate familiaritydata for person of interest step 300, which will be described in moredetail later with reference to FIG. 3.

In some embodiments, the familiar contexts data 380 is a histogram thatreflects the normalized frequency of occurrence for a set of predefinedscene contexts in a population of images associated with the person ofinterest 301. In this case, the compare image context to familiarcontexts step 213 can determine the context familiarity score 220 bysumming the frequencies from the familiar contexts data 380 histogramcorresponding to the scene contexts that are present in the imagecontexts data 212. This would result in a context familiarity score 220between 0.0 and 1.0. A context familiarity score 220 of 0.0 would resultif there was no commonality between the scene contexts represented inthe image contexts data 212 and the familiar scene contexts representedin the familiar contexts data 380. A context familiarity score 220 of1.0 would result if every element of the familiar contexts data 380 wasfound in the image contexts data 212.

In some embodiments, the familiar contexts data 380 can include acollection of predefined scene contexts. Each scene context in thecollection can have a score between 0.0 and 1.0, directly proportionalto the frequency of occurrence of each scene context in a population ofimages associated with the person of interest 301. In this case, a scenecontext that has the lowest frequency of occurrence can have a score of0.0, a scene context that has the highest frequency of occurrence canhave a score of 1.0, and the remaining scene contexts can have a scorebetween 0.0 and 1.0. In this case, the context familiarity score 220 fora digital image 201 can be the highest score in the familiar contextsdata 380 for those scene contexts that were identified for the digitalimage 201.

The second analysis path on the right side of FIG. 2 calculates theperson familiarity score 240. In a preferred embodiment, a detect facesstep 231 is used to detect the presence of any faces in the digitalimage 201. The output of the detect faces step 231 is a set of detectedfaces 232. In a preferred embodiment, the detect faces step 231 uses theface detector described by Schneiderman et al. in an article entitled“Probabilistic Modeling of Local Appearance and Spatial Relationshipsfor Object Recognition” (Proc. IEEE Conference on Computer Vision andPattern Recognition, pp. 45-51, 1998), which is incorporated herein byreference. This face detector implements a Bayesian classifier thatperforms maximum a posterior classification using a stored probabilitydistribution that approximates the conditional probability of face givenpixel data.

Next, a generate persons data step 233 automatically analyzes thedetected faces 232 to determine associated features which are stored asimage persons data 234. The image persons data 234 can be pixel data,features generated from the pixel data, facial models (e.g. an activeshape model) generated from the pixel data, metadata or tags associatedwith the detected faces 232, face recognition data for the detectedfaces 232 (i.e., the identities of the persons depicted by the detectedfaces 232), or any other features calculated from any combination of theabove. In some embodiments, the image persons data 234 can include alist of names of the persons who were identified from the detected faces232.

Once the image persons data 234 is obtained, a compare image persons tofamiliar persons step 235 compares the image persons data 234 topreviously calculated familiar persons data 390. The familiar personsdata 390 is generated in the generate familiarity data for person ofinterest step 300, and is a collection of information about personsfamiliar to the person of interest 301. In a preferred embodiment, thefamiliar persons data 390 includes the same kinds of information asfound in the image persons data 234. In some embodiments, the familiarpersons data 390 can also contain information about the relationshipbetween the persons whose data is contained in the familiar persons data390 and the person of interest 301 (e.g., self, relative, friend,acquaintance, etc.). In some embodiments, the familiar persons data 390can include information indicating the frequency of occurrence of thefamiliar persons in a collection of images associated with the person ofinterest 301. In some embodiments, the familiar persons data 390 caninclude information indicating the frequency of co-occurrence of sets ofpersons in a collection of images associated with the person of interest301 (see commonly-assigned U.S. Pat. No. 7,953,690 to Luo et al.,entitled “Discovering social relationships from personal photocollections,” which is incorporated herein by reference). Any of thesetypes of familiar persons data 390 can be used to provide an indicationof the degree of familiarity of a person to the person of interest 301.In some embodiments, the familiar persons data can also include datapertaining to celebrities (e.g., actors, musicians, politicians andhistorical figures) that would be familiar to the person of interest301.

The compare image persons to familiar persons step 235 matches dataelements from the image persons data 234 to those in the familiarpersons data 390. In a preferred embodiment, the compare image personsto familiar persons step 235 finds matches (or evaluates similarity)between the descriptions in the image persons data 234 and the familiarpersons data 390. The result of the compare image persons to familiarpersons step 235 is a person familiarity score 240. The personfamiliarity score 240 represents the overall familiarity of the faces inthe digital image 201 to the person of interest 301. In a preferredembodiment, the familiar persons data 390 includes informationpertaining to the relationship between the familiar person and theperson of interest 301. In this case, the person familiarity score 240can be determined responsive to this relationship. In some embodiments,the person familiarity score 240 can be determined responsive to thefrequency of occurrence of the familiar persons in a collection ofimages associated with the person of interest 301.

In a preferred embodiment, the person familiarity score 240 has a valuebetween 0.0 and 1.0. Based on research, it has been determined thatfaces which score the highest as being most familiar to the person ofinterest 301 are faces that are similar to the person of interest 301,faces that are similar to the faces of relatives, friends andcelebrities score the next highest, and faces of strangers score thelowest.

Techniques for detecting similarity values between faces (or betweenother types of image elements) are well known to those skilled in theart. The similarity value can represent a visual degree of similarity(i.e., how much do the image elements look like each other) or asemantic degree of similarity (i.e., how similar are the meanings of theimage elements). In some embodiments, a visual degree of similarity fortwo faces can be determined using the method described by Chopra et al.in the article “Learning a Similarity Metric Discriminatively, withApplication to Face Verification” (IEEE Conference on Computer Visionand Pattern Recognition, Vol. 1, pp. 539-546, 2005), which isincorporated herein by reference. Two faces can be said to be similar ifa similarity value is determined to exceed a predefined thresholdsimilarity value. Another method for determining the degree ofsimilarity between two objects that can be used in accordance with thepresent invention is described by Wang et al. in the article“Comparative object similarity for improved recognition with few or noexamples” (IEEE Conference on Computer Vision and Pattern Recognition,pp. 3525-3532, 2010), which is incorporated herein by reference.

An example of a simple scoring function for determining the personfamiliarity score 240 that reflects these trends is given by:

$\begin{matrix}{{PFS} = \frac{{W_{S} \times A_{S}} + {W_{R} \times A_{R}} + {W_{F} \times A_{F}} + {W_{C} \times A_{C}} + {W_{X} \times A_{X}}}{A_{S} + A_{R} + A_{F} + A_{C} + A_{X}}} & (1)\end{matrix}$where PFS is the person familiarity score 240, A_(S) is the area inpixels of all detected faces 232 that are similar to the person ofinterest 301, A_(R) is the area in pixels of all detected faces 232similar to a relative of the person of interest 301, A_(F) is the areain pixels of all detected faces 232 similar to a friend of the person ofinterest 301, A_(C) is the area in pixels of all detected faces 232similar to well-known celebrities, A_(X) is the area in pixels of alldetected faces that are not similar to any of these categories (e.g.,faces belonging to strangers to the person of interest 301), W_(S) is aweighting value for detected faces 232 that are similar to the person ofinterest 301, W_(R) is a weighting value for detected faces 232 that aresimilar to a relative of the person of interest 301, W_(F) is aweighting value for detected faces 232 that are similar to a friend ofthe person of interest 301, W_(C) is a weighting value for detectedfaces 232 that are similar to well-known celebrities and W_(X) is aweighting value for detected faces 232 that are considered to bestrangers. In some embodiments, W_(S)=1.0, W_(R)=W_(F)=W_(C)=0.7 andW_(X)=0.0. In other embodiments, different weighting values can be used.For example, weighting values of W_(S)=1.0, W_(R)=0.9, W_(C)=0.8,W_(F)=0.5, W_(X)=0.2 can be used to weight relatives more highly thancelebrities, which are in turn weighted more highly than friends andstrangers. In some embodiments, other categories of detected faces canalso be included as components in the person familiarity score 240, suchas “colleague” or “acquaintance,” which can each have an associatedweighting value.

In another embodiment, the person familiarity score 240 again has avalue between 0.0 and 1.0. In this case, the person familiarity score240 incorporates degree of similarity values which are used to weightsimilarity values determined between the detected faces and the personof interest 301. Based on the research, it has been determined thatincreasing the similarity of a detected face to the face of the personof interest 301 will render it more familiar to the person of interest301. In this case, the scoring function can be:

$\begin{matrix}{{PFS} = \frac{\sum\limits_{i = 1}^{N}\;{S_{i}A_{i}}}{\sum\limits_{i = 1}^{N}\; A_{i}}} & (2)\end{matrix}$where A_(i) is the area of the i^(th) detected face 232, S_(i) is asimilarity value (e.g., a value ranging from 0.0 to 1.0) providing anindication of the similarity of the i^(th) detected face 232 to the faceof the person of interest 301, and N is the number of detected faces232. If none of the detected faces 232 have any similarity to the faceof the person of interest 301 (or if there are no detected faces), thenPFS=0.0. If all of the detected faces 232 have maximum similarity to theface of the person of interest 301, then PFS=1.0.

It will be obvious to one skilled in the art that other variations ofthe metrics shown in Eqs. (1) and (2) can also be used to calculate theperson familiarity scores 240. For example, similarity values analogousto those used in Eq. (2) can be determined between the detected facesand the faces of known family members, friends and celebrities, orstrangers, and these similarity values can be combined with theweighting values in Eq. (1) rather than using a simple thresholdsimilarity value. For example, data for friends (A_(F) and W_(F)) andrelatives (A_(R) and W_(R)) can be used to calculate separate friendsfamiliarity scores (FFS) and relatives similarity scores (RFS),respectively, and a modified personal familiarity score PFS′ can bedetermined (e.g., PFS′=PFS+FFS+RFS). Additionally, the waiting values,the person familiarity scores, or both can be determined using, solely,or in combination with other methods, the evaluation/tracking of howfrequently the person of interest 301 visually interacts with the peopleand images of people whose faces are detected, including observing thesepeople in person, on TV, in pictures, or during various types oftelecommunication that use visual representation of these people'sfaces, such as via Skype, Facetime, or similar services.

After the context familiarity score 220 and person familiarity score 240have been calculated, a calculate interest level step 250 determines aninterest level 260 responsive to these values. The calculate interestlevel step 250 can use any appropriate function to combine the contextfamiliarity score 220 and the person familiarity score 240 to generatethe interest level 260. An indication of the context familiarity score220 will generally be stored in a processor-accessible memory, such aRAM associated with the processor used to implement the steps todetermine the interest level 260. In some embodiments, an indication ofthe interest level 260 can be stored as metadata in association with thedigital image 201 for use at a later time. The metadata can be stored inthe file used to store the digital image 201, or in another file that islinked in some way to the digital image 201.

Research conducted by the inventors, which is discussed in more detailbelow, has shown that the interest level of an image to a particularperson generally increases when the image contains faces that are morefamiliar to the particular person (i.e., an image of a relative would bemore interesting than an image of a stranger), and when the imageincludes scene contexts that are less familiar to the particular person(i.e., an image captured in an exotic location would be more interestingthan an image captured in the person's backyard). Accordingly, in apreferred embodiment, the interest level 260 determined by the calculateinterest level step 250 increases monotonically as the personfamiliarity score 240 increases and as the context familiarity score 220decreases. In the same preferred embodiment, the interest level 260determined by the calculate interest level step 250 decreasesmonotonically as the person familiarity score 240 decreases and as thecontext familiarity score 220 increases.

One such method for determining the interest level 260 (IL) is shown inthe following equation:IL=W _(PFS) ×PFS+W _(CFS)×(1−CFS)  (3)where PFS is the person familiarity score 240, CFS is the contextfamiliarity score 220, W_(PFS) is a weighting value for the personfamiliarity score 240 and W_(CFS) is a weighting value for the contextfamiliarity score 220. (In this example, it is assumed that PFS and CFSare normalized to have values between 0.0 and 1.0, although this is nota general requirement.) In some embodiments, W_(PFS)=W_(CFS)=0.5, whichplaces equal importance on the person familiarity score 240 and thecontext familiarity score 220 and provides a normalized maximum IL valueof 1.0. However, in other embodiments, non-equal weighting values can beused. For example, when multiple subjects (e.g., friends or relatives)are present in an image, their presence can outweigh the presence ofless interesting scene context, such that W_(PFS)>W_(CFS). Preferably,the weighting values, and even the form of the functional relationship,are determined experimentally based on interest levels determined for arepresentative population of observers. Additionally, the form of thefunctional relationship can be chosen so as to provide a sufficientdiscrimination between interest levels with respect to differentcontexts and persons depicted on the images. Eq. (1) can also use amodified personal familiarity score PFS′, where the constituentfamiliarity scores (e.g., PFS, FFS, RFS) have their own weightingfactors. The interest level 260 also does not need to be calculated witha normalized function.

As an alternative, the interest level 260 can be estimated with analternate version of Eq. (3) where the friends' familiarity scores(FFS), relatives familiarity scores (RFS), or other appropriatefamiliarity scores (e.g., celebrity familiarity scores) are broken outseparately from the personal familiarity score 240 (PFS). For example,this can be particularly useful if the weighting factors for thesevarious subjects of interest do not all scale linearly, which can occur,for example, when celebrity images are present.

FIG. 3 shows additional details for the generate familiarity data forperson of interest step 300 of FIG. 2 according to a preferredembodiment. A construct image collection step 302 is used to build acollection of images 303 by identifying image files related to theperson of interest 301. The image files may be stored in variouslocations, such as memory systems on user access devices 30 (FIG. 1) orother local machines or servers, on image servers 10 (FIG. 1) such asinternet image and video hosting websites (e.g., Shutterfly, Flickr andYouTube), or on social network servers 20 (FIG. 1) such as socialnetworking websites (e.g., Facebook, LinkedIn, Google+ and MySpace). Theimage files can include both digital still image files and digitalvideos. The collection of images 303 can include image files that belongto the person of interest 301, as well as image files that belong topeople socially connected to the person of interest 301 (e.g., relativesor friends that are connected through a social network).

A generate familiar contexts data step 400 is then used to determine thefamiliar contexts data 380 pertaining to the image contexts for thedigital images in the collection of images 303. The generate familiarcontexts data step 400 can use any method known in the art to determinethe familiar contexts data 380. In a preferred embodiment, the generatefamiliar contexts data step 400 uses the method that will be discussedin further detail with respect to FIG. 4.

The person of interest 301 has social connections to relatives, friends,and other people with whom they associate. This collection of peoplethat are associated with the person of interest 301 can be referred toas the person's “social network.” A discover social information step 311is used to gather social information 312 pertaining to the socialnetwork of the person of interest 301. The social information 312 caninclude genealogy information specifying the family relationshipsbetween the person of interest 301 and other individuals. It can alsoinclude lists of friends, colleagues or other individuals having anassociation with the person of interest 301.

The social information 312 can be gathered by many methods, such as byproviding a user interface that prompts the person of interest 301 todirectly provide the social information 312 (e.g., names of familymembers and friends). Social information 312 can also be collected bydiscovering the connections made by the person of interest 301 in onlinesocial networks (e.g., Facebook, LinkedIn, Google+ and MySpace).

In some embodiments, the discover social information step 311 candetermine the social information 312 based, at least in part, byautomatically analyzing the collection of images 303. For example, theaforementioned U.S. Pat. No. 7,953,690 to Luo et al. describes one suchmethod that can be used in accordance with the present invention. Thismethod involves analyzing the images in a photo collection to infersocial relationships between individuals pictured in the images.

A generate familiar persons data step 500 determines the familiarpersons data 390 pertaining to persons that are familiar to the personof interest 301. In a preferred embodiment, the familiar persons data390 is determined based on both the social information 312 and thecollection of images 303 using the method that will be described in moredetail with respect to FIG. 5. In other embodiments, the familiarpersons data 390 can be determined from only the social information 312,or only the collection of images 303.

In a preferred embodiment, the familiar persons data 390 includes facialdata pertaining to the appearance of each familiar person's face,together with familiarity data that provides an indication of the degreeof familiarity of each familiar person to the person of interest 301.

In some embodiments, the facial data in the familiar persons data 390can include pixel data representing faces of the familiar person,features determined from the pixel data, facial models (e.g. an activeshape model) generated from the pixel data, metadata or tags associatedwith the familiar persons, face recognition data for the familiarpersons, or any other features calculated from any combination of theabove.

It should be understood that the facial models used to enable thepresent invention can comprise one or more techniques known in the artsof facial recognition modeling. One applicable facial modelingtechnique, first described by Turk et al. in an article entitled“Eigenfaces for recognition” (Journal of Cognitive Neuroscience, Vol. 3,pp. 71-86, 1991), provides a 2-D model that is principally intended forassessing direct-on facial images. As another example of a facial modeluseful for recognizing people in images, the Active Shape Model (ASM) isa 2-D facial model in which faces are described by a series of facialfeature points. The ASM approach was described by Cootes et al. in apaper entitled “Active shape models—their training and application”(Computer Vision and Image Understanding, Vol. 61, pp. 38-59, 1995).Composite models, which extend facial recognition models to a 3-Dgeometry that map both the face and head, can also be useful foeenabling the present invention. A composite model approach has beendescribed by Blanz et al. in an article entitled “Face recognition basedon fitting a 3-D morphable model” (IEEE Transactions on Pattern Analysisand Machine Intelligence, Vol. 25, pp. 1063-1074, 2003).

In some embodiments, facial models can be determined for variousfamiliar persons (e.g., the person of interest 301 and friends andrelatives of the person of interest 301). The facial models can be thenbe provided as part of the familiar persons data 390. As the appearanceof subjects changes over time, it may be necessary to update theavailable facial models. This can be accomplished using methods andsoftware provided with the inventive system, or updated facial modelscan be determined externally and provided to this system. Methodsprovided in commonly-assigned U.S. Pat. No. 7,522,773, to Gallagher etal., entitled “Using time in recognizing persons in images” andcommonly-assigned U.S. Pat. No. 8,180,112, to Kurtz et al., entitled“Enabling persistent recognition of individuals in images”, both ofwhich are incorporated herein by reference, can be used for thispurpose.

In some embodiments, the familiarity data in the familiar persons data390 can include the names of the familiar persons, together with anindication of their relationship to the person of interest 301. Forexample, each familiar person can be categorized as “self,” “relative,”“friend,” “acquaintance,” “celebrity,” or “stranger.” In someembodiments, the familiarity data can include an indication of thefrequency of occurrence of the familiar persons in the collection ofimages 303. It can generally be assumed that persons that show up morefrequently in the collection of images 303 will be more familiar to theperson of interest 301.

FIG. 4 shows additional details for the generate familiar contexts datastep 400 of FIG. 3 according to a preferred embodiment. A determinescene contexts step 401 analyzes the collection of images 303 todetermine a set of corresponding scene contexts, which are representedusing scene contexts data 402. The determine scene contexts step 401 canuse any method known in the art to determine the scene contexts data402. In some embodiments, the determine scene contexts step 401 uses theone or more of the same analysis methods that were described earlierwith respect to the determine scene contexts step 211 in FIG. 2.

A calculate distribution step 403 determines statistics relating to thescene contexts data 402. In a preferred embodiment, the calculatedistribution step 403 determines the frequency of occurrence for each ofthe identified scene contexts in the scene contexts data 402. In thiscase, the familiar contexts data 380 is then used to store a list of thefamiliar scene contexts that occur in the collection of images 303,together with an indication of their frequency of occurrence.

FIG. 5 shows additional details for the generate familiar persons datastep 500 of FIG. 3 according to a preferred embodiment. A discoverpersonal relationships step 501 is used to determine personalrelationships data 502 pertaining to relationships between the person ofinterest 301 (FIG. 3) and other persons that are familiar to the personof interest 301. Preferably, the personal relationships data 502includes facial data pertaining to the appearance of each familiarperson's face, together with relationship data indicating a relationshipbetween the person of interest 301 and each familiar person.

In some embodiments, the personal relationships data 502 can beextracted from the social information 312, which was gathered fromsources such as social networking websites (e.g., Facebook andLinkedIn). For example, the social network for the person of interest301 can be analyzed to provide a list of familiar persons, together withtheir relationship to the person of interest 301. Images stored on thesocial networking website that have been tagged to contain the familiarpersons can then be analyzed to determine corresponding facial data. Forexample, methods described in commonly-assigned U.S. Patent ApplicationPublication 2011/0182482 to Winters et al., entitled “Method of personidentification using social connections,” which is incorporated hereinby reference, can be used to determine the facial data.

In some embodiments, the discover personal relationships step 501 candetermine the personal relationships data 502 by analyzing thecollection of images 303, for example by using the method described inthe aforementioned U.S. Pat. No. 7,953,690.

In some embodiments, the personal relationships data 502 of FIG. 5 canbe stored using a relational database. FIG. 6 shows one such example ofa relational database 100 which includes three tables that are used tostore the personal relationships data 502: a relationship connectionstable 101, a persons table 102 and a relationship types table 103. Thepersons table 102 includes a set of person records, each correspondingto an individual known to the person of interest 301, including theperson of interest 301 himself (i.e., “PERSON01”). The persons table 102has three fields. The first field, PERSON_ID, is a unique generatedidentifier for the person record. The second field, NAME (i.e., theperson's name) and the third field, FACE_DATA (i.e., data pertaining tothe face of the person), can be obtained from the social information 312and the collection of images 303 as described earlier. The combinationof the NAME and FACE_DATA fields can be used to identify any individualperson.

The relationship types table 103 is a collection of records, one foreach type of relationship that can be used in the compare image personsto familiar persons step 235 in FIG. 2. The relationship types table 103has two fields. The first field, TYPE, is a unique identifier for eachrelationship type. The second field, SCORING_INFO, can be used toassociate a level of familiarity with each relationship type. In someembodiments, the SCORING_INFO can be used by the compare image personsto familiar persons step 235 in FIG. 2 during the determination of theperson familiarity score 240. For example, in some embodiments, theSCORING_INFO field can be the weighting value applied to the facialareas in Eq. (1).

The relationship connections table 101 is used to store informationabout the relationships between pairs of people in the persons table102. The relationship connections table 101 has three fields. The firstfield, TYPE, matches the TYPE field of one of the records in therelationship types table 103. The second field and third fields,FIRST_PERSON and SECOND_PERSON, match the PERSON_ID fields of entries inthe persons table 102.

For example, the second entry in the relationship connections table 101can be interpreted as follows. There is a FAMILY relationship betweenPERSON01 (whose NAME is JEFFREY and whose FACE_DATA is <FACEDATA01>),and PERSON02, (whose NAME is JOANN and whose FACE_DATA is <FACEDATA02>).In this example, the corresponding SCORING_INFO (i.e., 0.8) can be usedas a weighting value in Eq. (1) for a facial area for a detected facethat is determined to match <FACEDATA02>.

Returning to a discussion of FIG. 5, a detect faces step 503 is used toanalyze the collection of images 303 to identify a set of detected faces504. Methods for detecting faces are well-known in the image analysisart, and any such method can be used in accordance with the presentinvention. In some embodiment, the detect faces step 503 can use thesame methods that were described earlier with respect to the detectfaces step 231 in FIG. 2.

A determine familiar persons data step 505 is then used to determine thefamiliar persons data 390. In a preferred embodiment, the detected faces504 are associated with corresponding familiar persons included in thepersonal relationships data 502. In some embodiments, the facial datastored in the personal relationships data 502 can be compared to thedetected faces 504 to determine the corresponding familiar persons.

As discussed earlier, in a preferred embodiment, the familiar personsdata 390 includes facial data pertaining to the appearance of eachfamiliar person's face, together with familiarity data that provides anindication of the degree of familiarity of each familiar person to theperson of interest 301. In some embodiments, the familiarity datainclude personal relationship information specified in the personalrelationships data 502. In some embodiments, the familiarity datainclude an indication of the frequency of occurrence of the familiarpersons in the collection of images 303. The frequency of occurrence canbe determined by counting the number of instances of the familiarpersons in the detected faces 504.

The illustrated embodiment of FIG. 5 shows the familiar persons data 390being determined responsive to both the personal relationships data 502and the detected faces 504. In other embodiments, the familiar personsdata 390 can be determined responsive to only one of the personalrelationships data 502 or the detected faces 504. For example, thedetermine familiar persons data step 505 can analyze the detected faces504 to count the number of times each unique face appears, without anyknowledge of the identity of each person, or their relationship to theperson of interest 301. In this case, the familiar persons data 390 cansimply include facial data pertaining to the unique faces, together withan indication of their frequency of occurrence. The assumption can thenbe made that the degree of familiarity of the persons will be related tothe corresponding frequency of occurrence in the collection of images303. In other cases, the personal relationships data 502 can be useddirectly as the familiar persons data 390, without any supplementalinformation provided by the detected faces 504.

The method for determining the interest level 260 of a digital image 201to a person of interest 301 that has been described with reference toFIGS. 2-6 has focused on determining the interest level 260 based on thefamiliarity of two particular image elements: the image context andpersons included in the image. Familiarity scores (context familiarityscore 220 and person familiarity score 240) are determined for each ofthese image elements, and the familiarity scores are then used todetermine the interest level 260. As such, the exemplary metric forinterest level 260 given in Eq. (3) can be expanded to include one ormore categories for special objects or image elements, measured with animage elements familiarity score EFS. Including the image elementsfamiliarity score EFS, as weighted by an image elements weighting factorW_(EFS), an exemplary expanded interest level 260 becomes:IL=W _(PFS) ×PFS+W _(CFS)×(1−CFS)+W _(EFS) ×EFS  (4)

The method of the present invention can be generalized to determine theinterest level 260 based on the familiarity of other types of imageelements. A flowchart of a generalized method for determining theinterest level 260 is shown in FIG. 7. A designate image elements step600 is used to automatically analyze the digital image 201 to identifyimage elements associated with the digital image 201, and to produceimage elements data 605, which provides an indication of the identifiedimage elements.

The image elements data 605 can include image contexts data 212 andimage persons data 234 as was described with reference to FIG. 2. Insome embodiments, the image elements data 605 can also include datapertaining to other types of image elements. For example, the imageelements data 605 can include data indicating the presence of certainobjects in the image. For example, objects can include animals (e.g.,pets or wild animals), consumer products (e.g., toys, clothing,electronics or vehicles), buildings (e.g., the house owned by the personof interest 301, public buildings located near the residence of theperson of interest 301 or famous buildings), landmarks (e.g., the Statueof Liberty, the Eiffel Tower or Mount Rushmore), food items (e.g.,packaged food items or prepared meals). The image elements data 605 canalso include information pertaining to various attributes of the digitalimage such as the distribution of colors, the coherence of coloredregions, image contrast or image sharpness.

The generate familiarity data for person of interest step 300 determinesthe familiarity of the relevant image elements to the person of interest301. The determined familiarity levels are represented in familiar imageelements data 610. As was discussed relative to FIG. 3, in a preferredembodiment, the familiarity of the image elements can be determined byanalyzing a collection of images 303 and social information 312associated with the person of interest 301. In some embodiments, thecollection of images 303 is analyzed to determine the frequency ofoccurrence of the image elements to provide an indication of the levelof familiarity of each image element. For example, it may be found thatobjects containing the team logo for a particular sports team occurfrequently in the collection of images 303. Therefore, it can beconcluded that when such image elements occur in other images they wouldbe very familiar to the person of interest 301.

A compare image elements to familiar elements step 615 determines imageelements familiarity scores 620 for the designated image elementsresponsive to the image elements data 605 and the familiar imageelements data 610. As was discussed relative to the compare imagecontext to familiar contexts step 213 and the compare image persons tofamiliar persons step 235 in FIG. 2, the familiarity score for aparticular designated image element can be determined based on datapertaining to the same or similar image elements in the familiarelements data.

The calculate interest level step 250 now determines the interest level260 based on the image elements familiarity scores 620. A functionalrelationship (interest level function 630) will generally be defined todetermine the interest level 260 as a function of the image elementsfamiliarity scores 620 (e.g., Eq. (4)). In some cases, the presence of afamiliar image element may have a positive correlation with interestlevel 260 (e.g., familiar persons). In other cases, the presence of afamiliar image element may have a negative correlation with interestlevel 260 (e.g., familiar scene contexts).

The interest level function 630 is defined using a define interest levelfunction step 625, which is usually performed at an earlier time. In apreferred embodiment, the define interest level function step 625defines the interest level function 630 by performing visual studieswith a representative population of observers. In some embodiments,different interest level functions 630 are determined for differentpopulation segments. In this case, representative populations ofobservers to be used in the visual studies can be selected to reflectcertain demographic attributes such as gender, age, ethnicity or culturethat are associated with each of the different population segments.

The following example illustrates a method for performing a visual studyto define the form of the interest level function 630. This exampleapplies to the embodiment described relative to FIG. 2 where theinterest level 260 is determined responsive to the context familiarityscore 220 and the person familiarity score 240.

In support of the development of the inventive method, an experimentalvisual study was performed to investigate how the familiarity of scenecontexts and depicted persons influence the perceived interestingness ofimages. A set of input digital images was selected for the study. Eachdigital image in the set contained a single person whose face wasvisible in the photograph, and a specific scene context such as a trainstation, an office or a birthday party. Two aspects of the familiarityof the input digital images were controlled as independent variables:“Person Familiarity” and “Scene Context Familiarity.”

The first independent variable, Person Familiarity, related to thefamiliarity of the person depicted in the digital image to theexperimental subject. Person Familiarity was a categorical variable withfour possible values: Self, Friend, Celebrity, and Stranger. The valuesare related to the degree of personal knowledge of the depicted personheld by an individual experimental subject. For example, a digital imagewith a Person Familiarity of “Self” would contain a person whose facewas similar to that of the experimental subject. Likewise, a digitalimage with a Person Familiarity of “Friend” would contain a person whoseface was similar to that of a friend of the experimental subject, and adigital image with a Person Familiarity of “Celebrity” would contain aperson whose face was similar to that of a celebrity who is well-knownto the experimental subject. A digital image with a Person Familiarityof “Stranger” would contain a face which is unknown to the experimentalsubject. Note that a digital image having a Person Familiarity of“Friend” for one experimental subject might have a value of “Stranger”for another experimental subject.

To provide a controlled level of Person Familiarity, the appearance ofan original face in the digital image was adjusted using a face morphingprocess. A face having an appearance similar to the experimental subjectwas obtained by using the face morphing process to modify the originalface in the image (who was a stranger to the experimental subject) bycombining it with the face of the experimental subject. Similarly, toprovide the Person Familiarity of “Celebrity” the face morphing processwas used to combine the original face in the image with the face of acelebrity, and to provide the Person Familiarity of “Friend” the facemorphing process was used to combine the original face in the image withthe face of a friend of the experimental subject. The Person Familiarityof “Stranger” was provided by retaining the unmodified face of thestranger in the original image. For this study, male faces were used formale subjects and female faces were used for female subjects.

The celebrity faces used for the face morphing process were chosen to behighly popular (but neutral on other aspects, such as beauty) female andmale celebrities. The face of the friend of the experimental subject waschosen to be the face of a work colleague. For a work colleague to beclassified as a “friend”, they had to satisfy three requirements: theyhad to meet with the subject regularly; they had to have at least onepoint of contact with the subject (e.g., a common work project); andthey had to be of the same gender.

The use of the face morphing process to provide the depicted facesenabled the study to investigate “familiarity” rather than “recognition”(i.e., the subjects should think the depicted face looks familiar tothem, rather than thinking that they know the person in the photo). Itwas found that 60% was a reasonable morphing level to express this senseof familiarity (i.e., the morphed face is interpolated to a point thatis 60% of the way from the original face in the image to the face ofinterest). The familiarity levels of celebrity faces, friend faces, andstranger faces in the original images that were used for morphing, wereverified in a post study questionnaire.

The second independent variable, “Scene Context Familiarity,” related tothe familiarity of the scene context of the digital image to theexperimental subject. Scene Context Familiarity was a categoricalvariable having two possible values, “Familiar” and “Unfamiliar.” Thevalues of Scene Context Familiarity varied according to the frequency ofthat scene context in a typical collection of photographs. To provide amore controlled selection of scenes having the Scene Context Familiarityvalues of “Familiar” and “Unfamiliar,” the familiarity levels of severalscene contexts were evaluated in a separate preceding experiment with adifferent group of participants, and only the scene contexts that wereconsistently rated among participants as having high and low familiaritywere selected. Scene contexts were selected to nominally beaesthetically neutralized or equivalent, as exemplified by images ofoffices, kitchens, train stations and gaming arcades, so that the impactof familiarity or unfamiliarity could be studied relative to perceivedinterestingness. For this study, it was assumed that the selected scenecontexts would have a similar familiarity level for all of theexperimental subjects. For example, an image captured in a kitchen wouldhave a higher familiarity to the experimental subjects, and an imagecaptured at a train station would have a lower familiarity to theexperimental subjects. The rationale for such an assumption was based onthe fact that the subjects shared similar living environment (greaterRochester, N.Y. region), and work place, and had relatively similareducation and income levels. (A t-test was performed based on apost-study questionnaire to confirm that the assumed scene contextfamiliarities for the selected images were indeed valid.) In otherstudies the image collections of individual experimental subjects couldbe analyzed to determine subject-specific scene context familiaritylevels.

A customized image set was prepared for each of the experimentalsubjects including 16 different images: 4 Person Familiarityvariations×4 different scene contexts (2 Unfamiliar scene contexts and 2Familiar). The dependent variable in the study was the “interestingness”or “interest level” of the images to the experimental subjects. Acustomized application was used to sequentially display the images in arandom order, and collect feedback from the experimental subjects. Auser interface 700 similar to that depicted in FIG. 8 was provided on asoft-copy display 720 to enable the experimental subjects to indicate alevel of interestingness for a displayed image 705. The user interface700 included a slide bar 710 that the subject could adjust to provide anindication of the relative level of interestingness for the displayedimage 705. A numerical value 715 was also displayed on the userinterface 700. The numerical value 715 ranged between 0 and 100,corresponding to the relative position of the control on the slide bar710. The numerical value 715 was continuously updated as the position ofthe slide bar 710 was adjusted by the experimental subject.

For the study, data for 22 male subjects and 20 female subjects wascollected. The median age for the experimental subjects was 52, with 23reporting that they were casual photographers and 19 reporting that theywere advanced photographers.

A repeated measures ANOVA test was conducted to examine the effect ofPerson Familiarity and Scene Context Familiarity on the ratedinterestingness scores of the digital images. Fixed factors in the modelincluded Person Familiarity, Scene Context Familiarity, gender andphotography expertise of the subject, quality of photo editing,technical and aesthetic photo quality scores, and perceived similarityof morphed faces. The subject's emotional state was added as a randomeffect. There were statistically significant main effects for both theScene Context Familiarity (F_(1, 624.2)=139.49, p<0.0001) and the PersonFamiliarity (F_(3, 625.3)=3.14, p<0.025), as well as a significantinteraction effect of (Scene Context Familiarity)×(Gender)(F_(1, 624.1)=13.16, p<0.0003). No significant interaction was found for(Person Familiarity)×(Scene Context Familiarity), as well as for (PersonFamiliarity)×(Gender). Photography expertise and emotional state of thesubject, and perceived similarity of the morphed face were insignificantcovariates. Quality of editing (F_(1, 631.8)=20.15, p<0.0001), aestheticquality (F_(1, 622.3)=6.71, p<0.0098) and technical photo quality(F_(1, 622.1)=4.75, p<0.0296) were seen to provide a significant, butsmall, contribution to interestingness scores.

FIG. 9A shows a graph of measured interestingness as a function of scenecontext familiarity for male and female subjects. It can be seen thatthere is a negative correlation between scene context familiarity andinterestingness—as the familiarity of the scene context increases, thecorresponding interest level to the observer decreases. This impliesthat observers are more interested in images captured in environmentsthat they do not commonly encounter than those captured in environmentsthat are familiar to them. This effect is somewhat stronger for malesubjects than for female subjects.

However, it should be understood that in some cases interestingness canpositively correlate with scene context familiarity for scenes havinghigh personal familiarity, although the described experiment did notexplore this connection. For example, context familiarity scores forpersonal settings, such as for images that depict portions of a givenindividual's own home or backyard landscape garden, can provide apositive correlation between scene context familiarity andinterestingness (interest level 260), due at least in part to anemotional connection of the viewer to the scene. Under suchcircumstances, scene context weighting can be larger than personalfamiliarity weightings, W_(CFS)>W_(PFS). Alternately, the scene contextweighting W_(CFS) can be modified or complemented by an emotionalresponse weighting factor (W_(ER)).

FIG. 9B shows a graph of measured interestingness as a function ofperson familiarity. It can be seen that there is a positive correlationbetween person familiarity and interestingness. Images containing peoplethat resembled the subject were most interesting, with images containingpeople that resembled celebrities being only slightly less interesting.The next most interesting were images containing people that resembledfriends, with images containing strangers being the least interesting.This leads to the conclusion that as the familiarity of the persondepicted in the image increases, the corresponding interest level to theobserver increases.

As noted previously, the experimental study deliberately used scenecontexts selected to nominally be aesthetically neutralized orequivalent (e.g., offices or kitchens). However, the inventive method isnot limited to such scene contexts, and it can be applied to analysis ofinterestingness for scenes that are aesthetically pleasing (e.g., grandnature scenes such as Yellowstone, or of a sunny flowery meadow, or ofplaces with graceful, intimate, or imposing architecture) or to scenesthat are aesthetically unpleasant (e.g., a junk yard or car wreck),whether these scenes are familiar or unfamiliar to the viewer.Accordingly, the scene context weighting W_(CFS) can be modified orcomplemented by an aesthetic response weighting factor (W_(AE)). Theaesthetic response weighting factor can have a low value (e.g.,W_(AE)≦0.1) for a very aesthetically unpleasant scene, and a high value(W_(AE)≧0.9) for a very aesthetically pleasant scene.

Similarly, in the prior discussions, interest level has been positivelycorrelated with facial recognition, whether the viewer is observingimages of people they know well (e.g., self, friends, relatives, orcelebrities), as compared to images of strangers, about whom interestlevels drop. While viewer interest in strangers is often low, there canbe exceptions. For example, iconic, artistic or provocative pictures ofstrangers can stimulate strong viewer interest. For example, the iconic1945 picture published in Life magazine, “V-J Day in Times Square” byAlfred Eisenstaedt, which depicts a sailor and nurse kissing in thestreet, typically elicits high viewer interest despite the fact thedepicted people are strangers. Although high interest pictures ofstrangers may not typically be present in personal photo collections,viewers can, for example, encounter them in advertising applications. Anaesthetic response weighting factor W_(AE) can be used to modifyestimates of interest level 260 for such circumstances.

The inventive method, as described herein, is distinct from existingmethods of evaluating image emphasis, appeal and degree of interest indigital images. For example, the method described in the aforementionedU.S. Pat. No. 6,671,405, to Savakis et al., uses information extractedfrom images, such as self-salient image features, including peoplerelated features (the presence or absence of people, the amount of skinor face area and the extent of close-up based on face size); objectivefeatures (the colorfulness and sharpness of the image); and subjectrelated features (the size of main subject and the goodness ofcomposition based on main subject mapping). The method of Savakis canalso use additional relative-salient features, such as therepresentative value of each image in terms of color content and theuniqueness of the picture aspect format of each image, and otherfeatures such as a variance in the main subject map. However, themethods of the prior art, including that of Savakis do not determinefamiliarity scores obtained from such information and other informationrelated to the person of interest 301. Therefore, unlike the presentinvention, such methods cannot provide a measure of interest level thatis specific to the individual viewer (i.e., person of interest 301).Consequently, the prior art method will rate images from the samecollection as having the same measure of emphasis, appeal and degree ofinterest, regardless of the viewer. The present invention, in contrast,can provide interest levels 260 specific to given persons of interest301.

The ability to determine an interest level 260 for a particular image toa person of interest 301 has many practical applications. One suchapplication is depicted in FIG. 10, which shows a flow chart of a methodfor selecting one or more digital images having a high interest levelfrom a set of candidate digital images 805 for display to person ofinterest 301.

As was discussed relative to FIG. 8, generate familiarity data forperson of interest step 300 is used to provide familiar image elementsdata 610 for the person of interest 301. Likewise, interest levelfunction 630 is defined using define interest level function step 625.

An identify candidate digital images step 800 is used to designate a setof candidate digital images 805 from which the images to be displayedwill be selected. In some embodiments the candidate digital images 805can be an image collection associated with the person of interest 301,or some subset thereof. In other embodiments, the candidate digitalimages 805 can be a set of images provided by some third party. Forexample, the candidate digital images 805 can be a set of advertisingimages for a particular product including different models and scenecontexts.

As was described relative to FIG. 8, designate image elements step 600is used to provide image elements data 605 for each of the candidatedigital images 805, and compare image elements to familiar elements step615 is used to determine corresponding image elements familiarity scores620.

A select candidate digital image(s) step 810 is then used to designateone or more selected digital image(s) 815 responsive to the imageelements familiarity scores 620 and the interest level function 630. Ina preferred embodiment, the select candidate digital image(s) step 810determines an interest level for each of the candidate digital images805 and designates one or more of the images having the highest interestlevels to be the selected digital image(s) 815. In some applicationsonly a single selected digital image 815 is provided. In otherapplications a plurality of selected digital images 815 are provided.

For example, the candidate digital images 805 can include fifty digitalimages associated with the person of interest 301. The interest levelsfor each of the candidate digital images 805 may range from highinterest levels (e.g., IL=0.9) for candidate digital images 805containing faces that are similar or identical to the face of the personof interest 301 (or to faces of friends and family of the person ofinterest 301 in an unfamiliar and compelling scene context, to lowinterest levels (e.g., IL=0.1) for candidate digital images 805 that donot contain any familiar faces and also lack highly familiar orcompelling scene contexts. The select candidate digital image(s) step810 can then select the twelve images having the highest interest levelfor use in a photo calendar.

In some embodiments, other factors besides the determined interestlevels can be used in the selection of the selected digital images 815.The additional factors can be included either as constraints, or ascomponents of a merit function. For example, in cases where a pluralityof selected digital images 815 are selected, it may be desirable toensure that the selected digital images 815 are not too similar to eachother. This could apply to the application where a photo calendar isbeing automatically created from the set of candidate digital images805. It is desirable to select images having a high interest level tothe person of interest 301, but it is also desirable that the selecteddigital images 815 not be too similar (i.e., it would be undesirable topopulate the calendar with twelve images of the person of intereststanding in front of the Eiffel Tower, even though each of these imageswould generally be found to have a very high interest level). For thisreason, a constraint can be added which requires that an appearancedifference between the selected digital images 815 is greater than apredefined threshold, that capture times associated with the selecteddigital images 815 differ by more than a predefined time interval orthat image capture locations associated with the selected digital images815 differ by more than a predefined distance. Examples of other factorsthat could be considered would include estimated image qualityattributes (e.g., sharpness, noise, colorfulness and facialexpressions/orientations and openness of eyes of depicted persons),image artifacts (e.g., red eye), image resolution, image orientation(i.e., landscape vs. portrait), and the number of different personsdepicted in the images.

In some embodiments, the select candidate digital image(s) step 810 canprovide a user interface that can be used to present the candidatedigital images having the highest determined interest levels to theperson of interest 301 (e.g., on a soft copy display). The userinterface can then include user controls to enable the person ofinterest 301 to select a subset of the presented images, or to accept orreject a proposed set of selected digital image(s) 815.

Once the set of selected digital image(s) 815 have been determined, adisplay selected digital image(s) step 820 is then used to display theselected digital image(s) 815 to the person of interest 301. In apreferred embodiment, the display selected digital image(s) step 820displays the selected digital image(s) 815 on a soft-copy display. Forexample, the selected digital image(s) step 820 can be displayed to theuser as a digital slideshow, used in an advertisement (e.g., on aninternet page or on a digital billboard), or presented to the user usingan appropriate user interface as suggested images for use in forming aphotographic product (e.g., a photographic enlargement, a photo collage,a photo calendar, a photo book, a photo T-shirt or a digital slideshowDVD). In other embodiments, the display selected digital image(s) step820 can display the selected digital image(s) 815 by printing them on adigital printer to provide a printed output that can be viewed by theperson of interest 301 (e.g., in a printed photographic product such asa photographic enlargement, a photo collage, a photo calendar or a photobook).

Another application of the interest level determination method isdepicted in FIG. 11, which shows a flow chart of an exemplary method formodifying one or more image elements in an initial digital image 830 toprovide a modified digital image having an increased interest level toperson of interest 301.

As was discussed relative to FIG. 10, generate familiarity data forperson of interest step 300 is used to provide familiar image elementsdata 610 for the person of interest 301. Likewise, interest levelfunction 630 is defined using define interest level function step 625.

The initial digital image 830 can be provided from a wide variety ofdifferent sources. For example, the initial digital image 830 can beselected from a collection of digital images associated with the personof interest 301, or it can be an advertising image that will bedisplayed to the person of interest 301. In a manner which is analogousto the method discussed in FIG. 10, the designate image elements step600 is used to provide image elements data 605 for the initial digitalimage 830. In some cases, the initial digital image 830 can be aphotographic image captured by a digital camera. In other cases, theinitial digital image 830 can be a computer generated image, or caninclude computer-generated image elements such as an avatar.

A modify image elements to increase interest level step 835 is used tomodify one or more image elements in the initial digital image 830 toprovide modified digital image 840 that has an increased interest levelto the person of interest 301, relative to the initial digital image830, as characterized by the interest level function 630.

There are a wide variety of different ways that the modify imageelements to increase interest level step 835 can modify the imageelements in the initial digital image 830 to increase the associatedinterest level. For example, in some embodiments, the image elementsdata 605 includes information relating to a depicted face in the initialdigital image 830. The modify image elements to increase interest levelstep 835 can modify the depicted face to replace it with the face of theperson of interest 301, or with the face of some other person that isfamiliar to the person of interest 301. Alternately, a face morphingprocess can be applied to the identified face to combine the depictedface with the face of the person of interest 301, or to combine it withthe face of some other person that is familiar to the person of interest301. In another scenario, an image of the person of interest 301 or animage of a person that is familiar to the person of interest 301 can beinserted into an image rather than just replacing an existing face. Thiscreates a new image element rather than simply modifying an existingimage element. Likewise, in some cases, an image element in the initialdigital image 830 can also be removed to increase the interest level.For example, the interest level of the image could be increased byremoving image elements associated with a familiar scene context. Ingeneral, the interest level of the image can be increased by addingfamiliar image elements that have a positive correlation with interestlevel and removing familiar image elements that have a negativecorrelation with interest level.

In some applications, the initial digital image 830 can include anavatar (e.g., a computer-generated representation of a person in a videogame). In this case, the avatar can be modified to give it features thatare similar to the person of interest 301, or to some other person thatis familiar to the person of interest 301. Some video gaming systemsinclude a built in video camera that captures images of the user. Thesecaptured images can be analyzed to determine facial informationassociated with the user that can be stored in the familiar imageelements data 610. The modify image elements to increase interest levelstep 835 can then use this information to modify the facial features ofthe avatar.

In the previous examples a depicted person in the initial digital image830 was modified to increase the interest level to the person ofinterest 301. Similarly, the scene context of the initial digital image830 can also be modified to increase the interest level to the person ofinterest 301. For example, if the initial digital image 830 includes aperson on a background associated with a very familiar scene context,the original background can be replaced with a new backgroundcorresponding to a scene context that is less familiar, and thereforemore interesting, to the person of interest 301.

In other applications, other types of image elements besides thedepicted persons and the scene context can be modified to increase theinterest level. For example, a pet in the initial digital image 830 canbe replaced with a pet that resembles the family pet of the person ofinterest 301. Alternatively, an image of the family pet can be insertedinto the image even if no pet was depicted in the initial digital image830.

It should also be understood that in some embodiments the exemplarymethod of FIG. 11 for providing a modified digital image 840 having anincreased interest level to person of interest 301, can alternately beadapted to decrease the interest level 260 in one or more images. All ofthe same techniques for changing images elements including faces, pets,objects, or scene context can be applied for this purpose.

Once the modified digital image 840 has been determined, a displaymodified digital image step 845 is then used to display the modifieddigital image 840 to the person of interest 301. In a preferredembodiment, the display modified digital image step 845 displays themodified digital image 840 on a soft-copy display. In other embodiments,the display modified digital image step 845 can display the modifieddigital image 840 by printing them on a digital printer to provide aprinted output that can be viewed by the person of interest 301.

FIGS. 12A-12C illustrate several scenarios where the various embodimentsof the present invention can be utilized. In some scenarios the presentinvention is used to provide images for use in printed photographicproducts such as photo collages, photo calendars and photo books. FIG.12A depicts an album page 900 that is automatically created for a photoalbum in accordance with the present invention. The photo album can be aprinted photo book that is provided to a customer, or can be a digitalphoto album that is adapted for viewing on a soft copy display (forexample, on a photo sharing website or a social networking website).

For example, consider the case where a customer desires to produce aphoto book including the best images from the past year. The customercan designate a set of candidate digital images 805 (FIG. 10)corresponding to the images in their image collection that were capturedin the past year. The process exemplary depicted in FIG. 10 can then beused to provide a set of selected digital images 815 corresponding tothe most interesting digital images in the set of candidate digitalimages 805. The number of selected digital images 815 can correspond tothe number of images that are needed to fill available locations inpredefined templates for the photo book pages. As discussed earlier,appropriate constraints can be applied during the selection process, forexample to select images having appropriate orientations and aspectratios, or to avoid selecting visually redundant images, or to avoidselecting a large number of images that were all captured during a shorttime interval or at the same geographic location.

In the depicted example album page 900, the template required twoimages, a first image 910 having a “portrait” orientation, and a secondimage 930 having a “landscape” orientation. For the first image 910, theselect candidate digital image(s) step 810 selected an image depictingthe person of interest 301 in front of the Eiffel Tower taken on aEuropean vacation. This image would have a high interest level to theperson of interest 301 since it contains a highly familiar person(himself) in an unfamiliar scene context (Paris). For the second image930, the select candidate digital image(s) step 810 selected anotherimage from the European vacation depicting the person of interest 301and an additional person 920 (the wife of the person of interest 301) infront of Big Ben taken on the European vacation. This image would have ahigh interest level to the person of interest 301 since it contains twohighly familiar people (his wife and himself) in an unfamiliar scenecontext (London). The other pages of the photo book would be populatedin a similar fashion.

In some applications, software is provided (e.g., on a photo sharingwebsite) that automatically populates the images in the photo book pagesin accordance with the methods for determining interest level for aperson of interest 301. A user interface can be provided to enable theuser to review the populated photo book pages and override any of theautomatic choices if they did not like some aspect of the selectedimages (for example, a facial expression of a depicted person).

In some embodiments, the user can be provided with a means to designatea person of interest 301 other than himself/herself. For example, theuser may desire to make a photo book that will be presented to theuser's mother as a birthday gift. In this case, the user's mother can bedesignated as the person of interest 301 so that the selected digitalimages 815 will be images having a high level of interest to the user'smother.

In some applications, the software can automatically populate the photobook pages without the user doing anything to initiate the process. Thephoto book can then be offered for sale to the user.

In other applications, the method described in FIG. 11 can be used tomodify one or more images in the set of candidate digital images 805 toprovide images for inclusion in the photo book.

FIG. 12B depicts an image display system 950 for presenting anadvertisement including a displayed digital image 980 that will have ahigh level of interest to a person 970 who is viewing the advertisement.The image display system 950 includes an image display 955 and a digitalcamera 960. The digital camera 960 has an associated field of view 965and is positioned to capture images of persons viewing the image display955. In some embodiments, the image display system 950 is a digitalbillboard system or a digital poster system that can be used to displayan advertisement in a public venue such as an airport, a mall, a retailstore, a restaurant or an amusement park.

It is desirable to present an advertisement that will have a high levelof interest to persons that are positioned to view the advertisement.Various embodiments of the present invention can be used to accomplishthis purpose. In one embodiment, a set of advertisements are prepared,each one including an advertising image with a different depicted person985. The depicted persons 985 in the set of advertisements can includemodels (including celebrities or strangers) having a variety ofdifferent appearance attributes (e.g., gender, skin color, facial shape,nose shape, mouth shape, eye color/shape, hair color/style, clothingstyles, jewelry styles and body art). The set of advertising images canbe used as the candidate digital images 805 and the person 970 can beused as the person of interest 301 in the embodiment depicted in FIG.10.

In some applications, the only information about the person of interest301 (i.e., person 970) will be one or more digital images captured bythe digital camera 960. In this case, the generate familiarity data forperson of interest step 300 (FIG. 10) can analyze the captured digitalimages to extract information about the appearance of the person ofinterest 301 (e.g., facial appearance data) which is provided asfamiliar image elements data 610 (FIG. 10). The compare image elementsto familiar elements step 615 (FIG. 10) can then compare the familiarimage elements data 610 to the image elements data 605 relating to theappearance of the depicted persons 985 in the set of advertisements todetermine image elements familiarity scores 620 (FIG. 10) that arerelated to the similarity between the person of interest 301 (i.e.,person 970) and the depicted persons 985. The select candidate digitalimage(s) step 810 (FIG. 10) can then provide a selected digital image815 (FIG. 10) corresponding to the advertisement that will have agreater level of interest to the person 970. The display selecteddigital image(s) step 820 (FIG. 10), can then display the selecteddigital image 815 on the image display 955 for viewing by the person970.

In some cases, there may be one or more additional persons 975 withinthe field of view 965 of the digital camera 960. In a preferredembodiment, the individual closest to the image display 955 can beselected to be the person of interest 301. In other cases, the imagescaptured by the digital camera 960 can be analyzed to determineadditional information that can be used to determine which of theindividuals should be treated as the person of interest 301 (e.g.,whether the individuals are looking at the image display 955).

In many cases, a group of persons who are in the field of view 965 ofthe digital camera 960 may be members of a family or may be a group offriends. Therefore, in some embodiments, one or more additional persons975 can also be used to provide familiar image elements data 610. Inthis case, the person 970 who is selected to be the person of interest301 can be assumed to have a person familiarity of “self” and theadditional persons 975 can be assumed to have a person familiarity of“friend.” In this way, the compare image elements to familiar elementsstep 615 can take into account the similarity between the depictedperson 985 and the additional persons 975 as well as between thedepicted person 985 and the person 970.

In some cases, the displayed digital image 980 can depict a plurality ofdepicted persons 985. In this case, the displayed digital image 980 canbe selected to have a high interest level to both the person 970 and theadditional person 975 by including one depicted person 985 having anappearance that is similar to the person 970 and a second depictedperson 985 having an appearance that is similar to the additional person975.

In some embodiments, the image display system 950 can perform imagerecognition to determine the identity of the person 970. For example, ifthe image display system 950 is used in a retail store which has adatabase of known customers that includes images of the customers, theimage of the person 970 can be compared to the images of the knowncustomers to determine the identity of the person 970. In otherembodiments, other means can be used to determine the identity of theperson 970. For example, an RFID signal can be detected from an ID cardassociated with the person 970. In this case, other information may beavailable relating to the person that be used to provide other types offamiliar image elements data 610 (e.g., home address information, facialinformation for family members, social relationship information and adatabase of products that the person 970 has purchased). This additionalinformation can be used by the compare image elements to familiarelements step 615 during the determination of the image elementsfamiliarity scores 620. For example, if the home address informationindicates that the person 970 lives in New York City, then a scenecontext for an advertising image having a scene context of “city” wouldgenerally have a higher familiarity level, but an advertising imagehaving a scene context of “mountains” would generally have a lowerfamiliarity level. It can generally be assumed that the familiaritylevel of a scene context associated with a particular geographiclocation would decrease as the distance from the home of the person 970increases. Therefore, in this example of a person 970 who lives in NewYork City, an advertising image having a scene context of “Statue ofLiberty” would generally have a higher familiarity level, but anadvertising image having a scene context of “Golden Gate Bridge” wouldgenerally have a lower familiarity level.

In some embodiments, demographic information (e.g., age, gender,ethnicity, culture) about the person 970 can be determined by analyzingthe digital images captured by the digital camera 960. This demographicinformation can then be used to infer familiar image elements data 610.For example, a male in his twenties would generally have differentfamiliarity with various scene contexts than a female in her sixties. Insome embodiments, a plurality of different sets of familiar imageelements data 610 can be predetermined for different demographicsegments. The demographic information determined for the person 970 canthen be used to select the set of familiar image elements data 610 thatis most appropriate for the person 970.

In other embodiments, the method described in FIG. 11 can be employed toprovide the advertising image to be used as the displayed digital image980 in FIG. 12B. For example, the appearance of the depicted person 985can be adjusted by using a face morphing process to combine the originalface of the model with the face of the person 970. The image of the faceof the person 970 that is used in the face morphing process can bedetermined from the images captured by the digital camera 960, or can bedetermined from a previously captured image (e.g., an image stored in aphoto ID database). In some embodiments, the final morphed image can bedetermined and substituted for the previously displayed digital image980. In other embodiments, a gradual morphing process can be used inwhich the original face in the displayed digital image 980 is graduallymorphed into a face that resembles the person 970 and the displayeddigital image 980 is continuously updated to show the intermediateimages.

Similarly, if the advertising image includes an avatar, the appearanceof the avatar can be adjusted accordingly to resemble the appearance ofthe person 970. Alternately, an image of the face of the person 970captured by the digital camera 960 can be used to replace the face ofthe depicted person 985.

FIG. 12C depicts another embodiment of an image display system 990 forpresenting a displayed digital image 995 that will have a high level ofinterest to person 970 who is viewing the displayed digital image 995.The image display system 990 includes an image display 955 and a digitalcamera 960. The digital camera 960 has an associated field of view 965and is positioned to capture images of persons viewing the image display955. In various embodiments, the image display system 950 can takedifferent forms, such as a personal computer having a web camera, atablet computer (e.g., an iPad) having a built in digital camera, ahandheld electronic device (e.g., a PDA or a smart phone) having a builtin digital camera, or a kiosk system in a retail store.

In some applications, the image display system 990 includes software foraccessing the internet and providing a displayed internet page 992 onthe image display 955. It is common for displayed internet pages 992 toinclude advertising content. In the illustrated example, the displayedinternet page 992 includes the displayed digital image 995 which is anadvertisement. As with the embodiment that was described relative toFIG. 12B, it will generally be desirable to present an advertisementthat will have a high interest level to the person 970. The same methodsthat were discussed relative to FIG. 12B can also be used for thisapplication to select an appropriate advertisement, or to modify one ormore image elements of an advertisement. In the illustrated example, thedisplayed digital image 995 is an advertisement that includes a depictedperson 985 which resembles the person 970, and is therefore highlyfamiliar, and has a scene context (Paris) that is unfamiliar to theperson 970. As discussed earlier, such images would be expected to havea high interest level to the person 970.

In many cases, the image display system 990 in FIG. 12C is a system thatis either owned by, or is closely associated with, the person 970 who isusing the system. In some embodiments, this fact can be leveraged by thegenerate familiarity data for person of interest step 300 (FIG. 10)during the process of generating the familiar image elements data 610(FIG. 10). For example, a collection of digital images stored on theimage display system 990, or on some server that can be accessed by theimage display system 990, can be analyzed to provide familiar imageelements data 610 as was described in FIG. 5.

In some applications, the displayed internet page 992 can be an internetpage for a social networking website (e.g., Facebook). In this case,social information 312 (FIG. 5) for the person 970 is readily availablefrom the network of connections with family and friends that have beenestablished by the person 970. In many cases, a collection of images 303(FIG. 5) will be stored on the social networking website that have beentagged to indicate the identities of faces contained in the images. Thisinformation can be utilized by the generate familiarity data for personof interest step 300 during the process of generating the familiarpersons data 390 (FIG. 5), which is used as a component of the familiarimage elements data 610.

In accordance with this scenario, when the person 970 accesses hisaccount on a social networking website, the displayed internet page 992provided by the social networking website for display on the imagedisplay 955 can include a displayed digital image 995 which is anadvertisement that is customized to have a high interest level to theperson 970. In this case, it is not necessary to use any image dataprovided by the digital camera 960 in order to determine the familiarimage elements data 610 since this data can be gleaned by analyzing thecollection of images 303 and social information 312 available on thesocial networking website.

In the examples just discussed with reference to FIGS. 12B and 12C,other data can be collected and used to complement the assessment ofviewer interest level provided by the present invention. For example,viewer time spent in front of the image display system 950 or 990viewing particular images, whether at a single viewing event or multipleviewing events, can be monitored or measured to provide complementarydata to about viewer interest levels. Data related to viewerinteractions, by gesture, speech, or user interface device (e.g., atouchpad or joystick) can also be valuable.

In other applications, the displayed digital image 995 can be anadvertisement presented as a component of a user interface for a videochatting application (e.g., Skype). In this case, familiar imageelements data 610 can relate to both the person 970, as well as theother person that the person 970 is chatting with. In some embodiments,a history of previous video chat sessions can also be used to provideinformation for the familiar image elements data 610. If two individualschat frequently, it can be inferred that they have a high degree offamiliarity.

In other applications, the displayed digital image 995 can be an elementof an educational presentation (e.g., an image included in aninteractive tutorial). In order to maximize the attentiveness of theperson 970 to the educational presentation, it is useful to providedisplayed digital images 995 having a high level of interest to theperson 970. Similarly, the displayed digital image 995 can be an elementof an entertainment presentation (e.g., an image included in aninteractive video game). In order to maximize the enjoyment of theperson 970 while they are viewing the entertainment presentation, it isuseful to provide displayed digital images 995 having a high level ofinterest to the person 970. The above-described methods are equallyapplicable for these scenarios.

In some embodiments, information about celebrities with whom the person970 is familiar can be determined by analyzing a usage history forvarious forms of digital media (e.g., songs that have been listened toon iTunes, videos that have been watched on YouTube or movies that havebeen watched on NetFlix).

In some embodiments, the familiar image elements data 610 can bepredetermined and stored on a memory system associated with the imagedisplay system 990 (for example, as a “cookie”). In this way, it doesnot need to be recalculated each time that the person 970 uses the imagedisplay system 990. The stored familiar image elements data 610 can beupdated as appropriate as new information becomes available about theperson 970 (e.g., if they add new persons to their social network).

The present invention may be employed in a variety of user contexts andenvironments. Exemplary contexts and environments include, withoutlimitation, wholesale digital photofinishing (which involves exemplaryprocess steps/stages such as: film input, digital processing, printsoutput), retail digital photofinishing (film input, digital processing,prints output), home printing (home scanned film or digital imagesinput, digital processing, prints output), desktop software (e.g.,software that applies algorithms to digital prints to make them better,or even just to change them), digital fulfillment (digital images input,digital processing, digital or hard copy output), kiosks (digital orscanned input, digital processing, digital or hard copy output), mobiledevices (e.g., camera, PDA or cell phone that can be used as aprocessing unit, a display unit, or a unit to give processinginstructions), and as a service offered via the World Wide Web.

In each context, the invention may stand alone or may be a component ofa larger system solution. Furthermore, human interfaces, input processesand output processes, can each be on the same or different devices andat the same or different physical locations, and communication betweenthe devices and locations can be via public or private networkconnections, or media based communication. Where consistent with theforegoing disclosure of the present invention, the method of theinvention can be fully automatic, may have user input (be fully orpartially manual), may have user or operator review to accept/reject theresult, or may be assisted by metadata (metadata that may be usersupplied, supplied by a measuring device (e.g. in a camera), ordetermined by an algorithm). Moreover, the algorithm(s) may interfacewith a variety of workflow user interface schemes.

A computer program product can include one or more non-transitory,tangible, computer readable storage medium, for example; magneticstorage media such as magnetic disk (such as a floppy disk) or magnetictape; optical storage media such as optical disk, optical tape, ormachine readable bar code; solid-state electronic storage devices suchas random access memory (RAM), or read-only memory (ROM); or any otherphysical device or media employed to store a computer program havinginstructions for controlling one or more computers to practice themethod according to the present invention.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   10 image server-   11 processor-   12 network interface unit-   13 non-volatile memory system-   14 volatile memory system-   15 database system-   20 social network server-   21 processor-   22 network interface unit-   23 non-volatile memory system-   24 volatile memory system-   25 database system-   30 user access device-   31 processor-   32 network interface unit-   33 non-volatile memory system-   34 volatile memory system-   35 input device-   36 display device-   37 image sensor unit-   40 communications connection-   50 image analysis system-   100 relational database-   101 relationship connections table-   102 persons table-   103 relationship types table-   201 digital image-   211 determine scene contexts step-   212 image contexts data-   213 compare image contexts to familiar contexts step-   220 context familiarity score-   231 detect faces step-   232 detected faces-   233 generate persons data step-   234 image persons data-   235 compare image persons to familiar persons step-   240 person familiarity score-   250 calculate interest level step-   260 interest level-   300 generate familiarity data for person of interest step-   301 person of interest-   302 construct image collection step-   303 collection of images-   311 discover social information step-   312 social information-   380 familiar contexts data-   390 familiar persons data-   400 generate familiar contexts data step-   401 determine scene contexts step-   402 scene contexts data-   403 calculate distribution step-   500 generate familiar persons data step-   501 discover personal relationships step-   502 personal relationships data-   503 detect faces step-   504 detected faces-   505 determine familiar persons data step-   600 designate image elements step-   605 image elements data-   610 familiar image elements data-   615 compare image elements to familiar elements step-   620 image elements familiarity scores-   625 define interest level function-   630 interest level function-   700 user interface-   705 displayed image-   710 slide bar-   715 numerical value-   720 soft-copy display-   800 identify candidate digital images step-   805 candidate digital images-   810 select candidate digital image(s) step-   815 selected digital image(s)-   820 display selected digital image(s) step-   830 initial digital image-   835 modify image elements to increase interest level step-   840 modified digital image-   845 display modified digital image step-   900 album page-   910 first image-   920 additional person-   930 second image-   950 image display system-   955 image display-   960 digital camera-   965 field of view-   970 person-   975 additional person-   980 displayed digital image-   985 depicted person-   990 image display system-   992 displayed internet page-   995 displayed digital image

The invention claimed is:
 1. A method, comprising: automaticallyanalyzing, by a processor, a digital image or metadata associated withthe digital image to designate one or more image elements in the digitalimage; determining familiarity levels of the designated image elementsto a particular person, wherein determining the familiarity levelscomprises: designating a digital image collection that is associatedwith the particular person, the digital image collection including aplurality of digital images; and determining a degree of similaritybetween each of the designated image elements and one or more imageelements in the digital images in the digital image collection, whereineach familiarity level is based on a corresponding determined degree ofsimilarity; determining an interest level of the digital image to theparticular person responsive to the determined familiarity levels; andstoring an indication of the determined interest level in a memory. 2.The method of claim 1, wherein determining the familiarity levelsfurther comprises: analyzing a frequency of occurrence of the designatedimage elements in the digital image collection; and determining thefamiliarity levels of the designated image elements responsive to thecorresponding frequency of occurrence.
 3. The method of claim 1, whereinthe digital image collection includes a personal image databaseassociated with the particular person or a set of digital imagesassociated with the particular person in an on-line social network. 4.The method of claim 1, wherein the degree of similarity is a visualdegree of similarity or a semantic degree of similarity.
 5. The methodof claim 1, wherein the image elements in the digital images in thedigital image collection are assigned familiarity scores, and whereinthe determination of the familiarity levels of the designated imageelements is also responsive to the assigned familiarity scores.
 6. Themethod of claim 5, wherein the familiarity scores are assigned based ontheir frequency of occurrence in the digital image collection.
 7. Themethod of claim 1, wherein the designated image elements include one ormore persons or objects depicted in the digital images.
 8. The method ofclaim 7, wherein the familiarity levels for depicted persons aredetermined responsive to social relationships between the depictedpersons and the particular person.
 9. The method of claim 7, wherein thefamiliarity levels for depicted persons are determined responsive toevaluating interactions between the persons and the particular personusing telecommunication processes.
 10. The method of claim 7, whereinthe depicted objects include animals, consumer products, buildings,landmarks or food items.
 11. The method of claim 7, wherein higherinterest levels are determined for digital images including depictedpersons or depicted objects having higher familiarity levels to theparticular person relative to the interest levels determined for digitalimages including depicted persons or depicted objects having lowerfamiliarity levels to the particular person.
 12. The method of claim 1,wherein the image elements include scene contexts.
 13. The method ofclaim 12, wherein the scene contexts comprise at least one ofsurroundings, locations, events, activities, scene classifications, andattributes associated with the digital image.
 14. The method of claim12, wherein higher interest levels are determined for digital imagesincluding scene contexts having lower familiarity levels to theparticular person relative to the interest levels determined for digitalimages including scene contexts having higher familiarity levels to theparticular person.
 15. The method of claim 1, wherein the determinationof the interest level is responsive to a gender, age, ethnicity orculture of the particular person.
 16. A non-transitory computer readablemedium having instructions stored thereon, the instructions comprising:instructions to automatically analyze a digital image or metadataassociated with the digital image to designate one or more imageelements in the digital image; instructions to determine familiaritylevels of the designated image elements to a particular person, whereindetermining the familiarity levels comprises: designating a digitalimage collection that is associated with the particular person, thedigital image collection including a plurality of digital images; anddetermining a degree of similarity between each of the designated imageelements and one or more image elements in the digital images in thedigital image collection, wherein each familiarity level is based on acorresponding determined degree of similarity; instructions to determinean interest level of the digital image to the particular personresponsive to the determined familiarity levels; and instructions tostore an indication of the determined interest level in a memory. 17.The non-transitory computer medium of claim 16, wherein determining thefamiliarity levels further comprises: analyzing the frequency ofoccurrence of the designated image elements in the digital imagecollection; and determining the familiarity levels of the designatedimage elements responsive to the corresponding frequency of occurrence.18. A system, comprising: a memory; and one or more processors coupledto the memory, wherein the one or more processors are configured to:automatically analyze a digital image or metadata associated with thedigital image to designate one or more image elements in the digitalimage; determine familiarity levels of the designated image elements toa particular person, wherein to determine the familiarity levels the oneor more processors are configured to: designate a digital imagecollection that is associated with the particular person, the digitalimage collection including a plurality of digital images; and determinea degree of similarity between each of the designated image elements andone or more image elements in the digital images in the digital imagecollection, wherein each familiarity level is based on a correspondingdetermined degree of similarity; determine an interest level of thedigital image to the particular person responsive to the determinedfamiliarity levels; and store an indication of the determined interestlevel in the memory.
 19. The system of claim 18, wherein to determinethe familiarity levels e one or more processors are further configuredto: analyze the frequency of occurrence of the designated image elementsin the digital image collection; and determine the familiarity levels ofthe designated image elements responsive to the corresponding frequencyof occurrence.
 20. The system of claim 18, wherein the digital imagecollection includes a personal image database associated with theparticular person or a set of digital images associated with theparticular person in an on-line social network.